Systems and methods for automatic test generation

ABSTRACT

Systems and methods for generating and evaluating driving scenarios with varying difficulty levels is provided. The disclosed systems and methods may be used to develop a suite of regression tests that track the progress of an autonomous driving stack. A robustness trace of a temporal logic formula may be computed from an always-eventually fragment using a computation graph. The robustness trace may be approximated by a smoothly differentiable computation graph, which can be implemented in existing machine learning programming frameworks. The systems and methods provided herein may be useful in automatic test case generation for autonomous or semi-autonomous vehicles.

CROSS-REFERENCE TO RELATED APPLICATION

This application claims priority to U.S. Provisional Application No.62/746,700, filed on Oct. 17, 2018, the contents of which are herebyincorporated by reference in its entirety.

FIELD OF TECHNOLOGY

Aspects of the present disclosure relate to simulation data and testcase generation, more particularly to systems and methods fortranslating signal temporal logic formulas into computation graphs, foruse in complex machine learning.

BACKGROUND

The “rules of the road” govern how people behave while driving.High-level behaviors may often be described using rule-based orlogic-based techniques. This technique, however, depends on domainknowledge, and not all the rules are always met as there is a spectrumin how individuals may obey these rules. For example, some drivers tendto tailgate, while others maintain a three-second gap.

Signal Temporal Logic (STL) is an expressive language to describesignals, both continuous and discrete. STL has been adopted as aspecification language for discrete, continuous, and hybrid systems,including robots and autonomous vehicles. More recently, STL formulashave been used for clustering and classification of time-varying traces,providing an explainable form machine learning. However, prior systemsand methods rely on computationally expensive algorithms that can onlyhandle monotonic STL formulas, i.e. formulas that become monotonicallymore true or less true as a parameter is varied.

SUMMARY

Aspects of the present disclosure provide a testing strategy forautonomous vehicles that make use of rule-based domain knowledge. Amodeling language is provided that offers rigidity with regards to therule-based nature of driving, but also some flexibility to describe thespectrum in which these rules may be followed. Aspects of the presentdisclosure further provide systems and methods for generating drivingscenarios with varying difficult levels in order to develop a suite ofregression tests that track the progress of an autonomous driving stack.In order to certify the safety of a cyber-physical system, the systemmust be strenuously stress-tested in a diverse range of scenarios toensure the system is certifiably safe and robust, i.e., failure occurswithin some ϵ-probability. According to aspects of the disclosure arobustness trace of a temporal logic formula may be computed from analways-eventually fragment using a computation graph. By using arelaxation, the robustness trace may be approximated by a smoothlydifferentiable computation graph, which can be implemented in existingmachine learning programming frameworks. The systems and methodsprovided herein may be useful automatic test case generation forautonomous or semi-autonomous vehicles.

According to one aspect of the present disclosure, a method ofevaluating a test case is provided. The method may include receiving atimed trace input and generating a parametric signal temporal logicformula representative of a driving scenario. The parametric temporallogic formula may be solved for at least one parameter, wherein thevalue of the at least one parameter provides an optimal description ofthe timed trace. The test case may be evaluated using the value of theat least one parameter.

According to another aspect of the present disclosure, a system forevaluating a testing scenario is provided. The system may include amemory and at least one processor coupled to the memory. The processormay be configured to receive a timed trace input, generate a parametricsignal temporal logic formula representative of a driving scenario, andsolve the parametric temporal logic formula for at least one parameter,wherein the value of the at least one parameter provides an optimaldescription of the timed trace. The processor may further be configuredto evaluate the test case using the value of the at least one parameter.

According to another aspect of the present disclosure, a non-transitorycomputer-readable medium having program code recorded thereon forevaluating a testing scenario is provided. The program code may beexecuted by a processor and comprise program code to receive a timedtrace input, program code to generate a parametric signal temporal logicformula representative of a driving scenario, and program code to solvethe parametric temporal logic formula for at least one parameter,wherein the value of the at least one parameter provides an optimaldescription of the timed trace. The non-transitory computer-readablemedium may further include program code to evaluate the test case usingthe value of the at least one parameter.

This has outlined, rather broadly, the features and technical advantagesof the present disclosure in order that the detailed description thatfollows may be better understood. Additional features and advantages ofthe present disclosure will be described below. It should be appreciatedby those skilled in the art that this present disclosure may be readilyutilized as a basis for modifying or designing other structures forcarrying out the same purposes of the present disclosure. It should alsobe realized by those skilled in the art that such equivalentconstructions do not depart from the teachings of the present disclosureas set forth in the appended claims. The novel features, which arebelieved to be characteristic of the present disclosure, both as to itsorganization and method of operation, together with further objects andadvantages, will be better understood from the following descriptionwhen considered in connection with the accompanying figures. It is to beexpressly understood, however, that each of the figures is provided forthe purpose of illustration and description only and is not intended asa definition of the limits of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The features, nature, and advantages of the present disclosure willbecome more apparent from the detailed description set forth below whentaken in conjunction with the drawings in which like referencecharacters identify correspondingly throughout.

FIG. 1(a) depicts a graphical representation of an STL formula accordingto an aspect of the present disclosure.

FIG. 1(b) depicts a flow diagram of a computation of robustness androbustness trace according to an aspect of the present disclosure.

FIG. 2 depicts a parse tree and a computation graph according to anaspect of the present disclosure.

FIGS. 3(a)-(c) depict computation graph representations of therobustness function for predicate and Boolean connectives according toan aspect of the present disclosure.

FIG. 4 depicts a computation graph for a recurrent network for computingthe robustness of signal temporal logic formulas according to an aspectof the present disclosure.

FIG. 5 depicts a computation graph according to an aspect of the presentdisclosure.

FIG. 6 depicts a flow diagram of a method of solving a parametric signaltemporal logic using a computational graph and a differentiableapproximation according to an aspect of the present disclosure.

FIG. 7 depicts a graphical representation of a traffic weaving scenarioaccording to an aspect of the present disclosure.

FIG. 8(a)-(b) depict results of a time to collision simulation accordingto one aspect of the present disclosure.

FIG. 9 depicts a hardware implementation for a systematic testing systemaccording to aspects of the present disclosure.

DETAILED DESCRIPTION

The present disclosure generally relates to systems and methods forgenerating driving scenarios by computing robustness traces of signaltemporal logic formulas using computation graphs. The robustness tracemay be approximated by a smoothly differentiable computation graph,which may be implemented in a machine-learning framework, particularlyone suited for autonomous or semi-autonomous driving scenarios.

According to an aspect of the present disclosure, a method forconstructing a computation graph to evaluate the robustness of an STLformula is presented. The computation graph may be equivalent todirectly computing the robustness recursively. According to anotheraspect, differentiable approximations may be made to the robustnessformulas to show that a parametric STL (pSTL) problem may be efficientlysolved using backpropagation on the computation graph. According toanother aspect, the method may be implemented in a system and used toinvestigate driving scenarios, such as a lane-changing or merging, forsystematic scenario-based testing for autonomous driving. The disclosedsystems and methods may be used to efficiently learn pSTL parameters,use them to gain interpretability into the system, and synthesize thisinformation to generate new and more challenging test scenarios in orderto improve the testing and validation of safety-critical systems.

Formal methods are a mathematically rigorous, logic-based techniquesused for specification, design and verification of software and hardwaresystems. They provide a means to symbolically examine a system andestablish correctness or safety properties. In particular, STL is anexpressive specification language that can describe properties of bothdiscrete-time and real-valued dense-time signals. This capability makesit a versatile specification language for cyber-physical systems. Inaddition to Boolean semantics, STL is equipped with quantitativesemantics. STL can provide a robustness value—a continuous real-valuedscalar that represents the degree of satisfaction over thespecification. Recently, STL has gained popularity because it can beapplied to continuous and hybrid dynamics, and be used to describelong-term behaviors of those systems. For example, STL has been used asconstraints in model predictive control and stochastic control problems,and to describe long-term goals in reinforcement learning.

The inverse problem, i.e., constructing an STL specification given asignal, presents an interesting yet challenging problem. It isinteresting because learning STL formulas from complex time-series datacan offer valuable insight and interpretability into the structure andproperties of the system. It may also potentially improve other methodsthat rely on the signal, such as classification, prediction, anddecision-making. Constructing and STL specification from a signal,however, may be challenging because the space of STL formulas isinfinitely large and the solution is not unique. Previous systemspropose methods to mine STL specifications from time-series data and useit for classification and prediction. For computational tractability,however, those methods limit the class of formulas that can be used. Onecommon approach is parametric STL (pSTL). A pSTL formula is a templateSTL formula, and given time series data, it may be possible to optimizethe formula over the parameters to find the values that provide thetightest fit in terms of robustness. This parameter identification canbe thought as a form of feature extraction from time series data. A keydrawback of existing techniques, however, is that they only supportmonotonic formulas, i.e. formulas whose satisfaction dependsmonotonically on the parameters.

Aspects of the present disclosure present a system and method that usescomputation graphs and smooth differentiable approximations to tractablyand efficiently learn parameters of complex, possibly non-monotonic,pSTL formulas that contain ‘Always’ and ‘Eventually’ temporal operators.By using computation graphs, state-of-the-art machine learningcomputation tools may be leveraged. These state-of-the-art machinelearning tools provide auto-differentiation, which allows the system toefficiently learn the parameters of a pSTL formula. In addition, thesetools afford various other computational advantages, such as ease ofparallelization and fast optimization routines.

Another popular research field is machine learning, and more recently,deep learning. Machine learning has been very successful in manydomains, such as natural language processing, computer vision, andend-to-end learning for robotic systems. To accommodate this surge,there has been industrial efforts in developing very efficientcomputational infrastructure (e.g., PyTorch and Tensorflow) for machinelearning techniques. Since deep learning algorithms rely on large-scalecomputation graphs and backpropagation, these tools are in fact a verygeneral and highly optimized tool for building arbitrarily complexcomputation graphs. Representing STL robustness formulas as computationgraphs takes advantage of the benefits of using state-of-the-art machinelearning tools.

A major drawback in many deep learning algorithms however, especiallythose involving dense time-series data, is that the learned model isoften extremely opaque and thus difficult to interpret. It is notobvious how to inject prior knowledge into the network in a groundedway, and it is challenging to properly encode desired long-termbehaviors into an otherwise dense neural network. Deep learning has beenused in many safety-critical systems (e.g., autonomous driving, computervision, and manipulation) but due to its black-box nature, validatingthe safety of such systems is extremely difficult.

According to one aspect of the present disclosure, a system and methodmay translate STL formulas into computation graphs, i.e., the same“language” used by modern machine learning tools. The translationbridges the gap between STL, a type of formal method used for safetyverification and model checking of time-series data, and deep learning,a very powerful function approximator that is used in manysafety-critical systems. Aspects of the present disclosure incorporatelogical structure within deep learning frameworks to build a powerfulyet logically grounded model that is applicable to many domains.

As artificial intelligence (AI) systems are given more responsibilitiesin society, it is important to ensure that they are making decisions forthe right reasons and be able to justify their actions. The currentdominant class of deep learning AI algorithms are notoriously opaque. Somuch so, in fact, governments have requested and sought out solutionsand techniques that can provide legal and ethical explanations, whichare important for a wide range of safety-critical applications,including self-driving cars, military systems, and semi-automatedwarehouses.

For instance, deep learning has found success in the field of computervision but many of the core techniques rely on extremely dense andopaque networks, making it difficult to justify the image classificationresult. Recent systems have been aimed at generating explanations for anobject identification decision. For example, some prior systems aim togenerate text and/or identify areas in an image as a form explanation.These methods however rely on a separate neural network to generateexplanations, and are separate from the decision-maker itself.

Although these approaches help explain the classification result, it isnot immediately apparent as to how this can be used to ensure safety foran AI system. Alternatively, in the field of controls and planning,Inverse Reinforcement Learning (IRL) is a machine learning techniquethat injects interpretability by learning a reward function thatrepresents the system's incentives (e.g., human's preference). Althougheffective in learning the system's preference, is it highly biased bythe choice of features. For larger and more complex systems, thesefeatures are often learned via neural networks and thus becomeuninterpretable.

Logical formulas, on the other hand, are a uniquely powerful tool forgenerating explanations that can be used as part of a justification andmay be used for control synthesis to ensure the system satisfies thespecification. Logical formulas may also be a lens for one to gainbetter insight into the problem at hand. The inverse problem, however,i.e., learning logical formulas from data, may also be extremelypowerful as it offers insight into the structure and logical propertiesof an independent system. This may be a very challenging problem andoften leads to intractability. Tractable methods may limit the class oflearnable formulas.

A proxy for logical structure is to use decision trees, in the sensethat logical predicates are tested for truth or falsehood at each nodeof the tree. Prior methods provide a differentiable formulation fordecision trees that can be trained by backpropagation. Decision trees,however do not have an internal state, which means that they do notprovide the same dynamic richness as many temporal logic formulas (e.g.,STL), which can express and monitor properties that evolve over time.

A middle ground between tractability and using the powerful capabilitiesof STL may be to use pSTL. Prior methods rely on algorithms that arerestricted to monotonic formulas. Although monotonic formulas representa broad class of STL methods, this requirement may be relaxed and a muchbroader framework for solving general pSTL problems may be developed.

Aspects of the present disclosure provide a computation graphrepresentation of STL formulas and, using a relaxation, pSTL formulasmay be optimized using machine learning training frameworks. In thisway, the gap between the efficient trainability of classical machinelearning algorithms and the argumentative strength of logical formulasmay be bridged.

According to an aspect of the present disclosure, STL and its parametricextension, pSTL may be defined as the following:

-   -   Definition 1 (Timed Trace): A timed trace s may be an ordered        finite sequence of states and their associated time, s=(x₀, t₀),        . . . , (x_(n), t_(n)) where t_(i−1)<t_(i) and x_(i) ∈        ^(n). Further, the following notation made be used:        s(t_(i))=x_(i).    -   Definition 2 (Timed Trace Subsequence). Given a timed trace s        that starts at t₀, a timed trace subsequence of (s, t_(i)) is a        timed trace starting at time t_(i), i∈        ₀. That is,

(s,t _(i))=(x _(i) ,t _(i)), . . . ,(x _(n) ,t _(n))

Further, we use the following notation s_(i) (s, t_(i)).

Additionally, the assumption is made that the time steps are uniformlyspaced.

According to aspects of the present disclosure, temporal logic and itsmany variants for example, Linear Temporal Logic (LTL), Bounded LinearTemporal Logic (BLTL), Linear Temporal Logic over finite traces (LTLf),and Metric Temporal Logic (MTL), may be formalisms for reasoning abouttimed traces. They may be powerful tools to express specifications, andenable a wide array of assurance techniques, such as formalverification, falsification, model checking, and systematic testing.These logical languages may be used in many domains including robotmotion planning and control, biological systems, and network planning.Specifically, STL, an extension to Linear Temporal Logic (LTL), is aspecification language for real-valued signals and can be applied tomany continuous and hybrids systems.

STL formulas may be defined over predicates of the form f(s)<c, where sis a timed trace (signal), f:

^(n)→

is a function and c ∈

. STL formulas are written using the following grammar:

I:=(a,b)|(a,b]|[a,b)|[a,b]

ϕ:=true|f(s)<c|¬ϕ|ϕ∧ψ|ϕ∨ψ|⋄ _(Iϕ)|□_(Iϕ) |ϕU _(Iψ) |ϕT _(Iψ)

where f(s)<c is a predicate,

(negation/not), ∧ (conjunction/and), and ∨ (disjunction/or) are Booleanconnectives, and ⋄ (eventually), □ (always), U (until) and T (then) aretemporal operators. f(s)<c may be achieved by applying a negation. Thetemporal operators may have an associated time interval I where 0≤a<b.For ease of notation, I may be dropped from the grammar when a=0, b=∞.Further the following assumption may be made:

Assumption 1. Without loss of generality, I may be only considered to beof the form [a, b] or [a, ∞] where 0≤a<b. By the assumption that thetime steps are uniformly spaced, if I does not include endpoints, theboundaries may be changed to the nearest inclusive time step.

Let (s, t) be a timed trace starting at time t. The Boolean semantics ofSTL may be defined recursively:

(s,t)|=f(s(t))<c ⇔f(s(t))<c

(s,t)|=¬ϕ ⇔¬((s,t)|=ϕ)

(s,t)|=ϕ∧ψ ⇔((s,t)|=ϕ)∧((s,t)|=ψ)

(s,t)|=ϕ∨ψ ⇔((s,t)|=ϕ)∨((s,t)|=ψ)

(s,t)|=⋄_(Iϕ) ⇔∃t′∈I⊕t s.t.(s,t′)|=ϕ

(s,t)|=□_(ϕ) ⇔∀t′∈I⊕t s.t.(s,t′)|=ϕ

(s,t)|=ϕU _(Iψ) ⇔∃t′∈I⊕t s.t.(s,t′)|=ψ)∧((s,t)|=□_([0,t′])ϕ)

(s,t)|=ϕT _(Iψ) ⇔∃t′∈I⊕t s.t.(s,t′)|=ψ)∧((s,t)|=⋄_([0,t′])ϕ)

For a timed trace (s, t) starting at time t, satisfying □ϕ means ϕ isalways true for the entire sequence (since the I is dropped, I=[0, ∞)).While satisfying ⋄ϕ means at some time along the sequence, ϕ is true atleast once. Since STL specifications are defined recursively, temporaloperators may be composed with each other. For example, a timed trace(s, t) satisfying ⋄□_([1,3])ϕ means that eventually ϕ will always besatisfied over the next 1-3 time units.

Since STL formulas are defined recursively, STL may be represented usinga parse tree where each node represents an operation, and the leaves arethe predicates. The order of operations for a formula ϕ may be denotedby O_(ϕ).

-   -   Definition 3 (STL Ordering): For an STL formula ϕ, let PT be the        parse tree of ϕ. Define O_(ϕ)={φ₁, φ₂, . . . φ_(k)} as the        post-order traversal of PT.

Example 1

Let ϕ=⋄_(I) ₁ (f(s)<c₁∧g(s)<c₂)∨□_(I) ₂ h(s)<c₃. Then O_(ϕ)={f(s)<c₁,g(s)<c₂, ∧, ⋄_(I) ₁ , h(s)<c₃, □_(I) ₂ , ∨}.

A special property of STL is the notion of a robustness degree, a realvalue that describes how much a timed trace satisfies (or fails tosatisfy) an STL specification. For example, if a signal does not satisfyan STL specification, the robustness degree will indicate the level ofviolation. This robustness metric may expand the utility of temporallogic in many continuous domains such as in problems pertaining toclassification and prediction, optimal control, Hamiltonian-Jacobianreachability, and reinforcement learning.

The robustness degree can be calculated recursively according to thefollowing quantitative semantics:

ρ(s,t,true)=ρ_(max)

ρ(s,t,f(s)<c)=c−f(s(t))

p(s,t,¬ϕ)=−ρ(s,t,ϕ)

p(s,t,ϕ∧ψ)=min(ρ(s,t,ϕ),ρ(s,t,ψ)

p(s,t,ϕ∨ψ)=min(ρ(s,t,ϕ),ρ(s,t,ψ)

ρ(s,t,⋄ _(Iϕ))=(ρ(s,t′,ϕ)

ρ(s,t,□ _(Iϕ))=(ρ(s,t′,ϕ)

ρ(s,t,ϕU _(Iψ))=(min(ρ(s,t′,ψ),ρ(s,t″,ϕ)))

ρ(s,t,ϕT _(Iψ))=(min(ρ(s,t′,ψ),ρ(s,t″,ϕ)))

Further, there is no difference in using a strict, or non-strictinequality, as the robustness value does not change.

A robustness trace, according to one aspect of the disclosure may bedefined:

-   -   Definition 4 (Robustness Trace): Given a timed trace s starting        at time t₀ and an STL formula ϕ, the robustness trace τ(s, t₀,        ϕ) is a sequence of robustness values for each subsequence s_(i)        of s. That is:

$\begin{matrix}{{{\tau \left( {s,t_{0},\varphi} \right)} = \tau_{0}},\tau_{i},{\ldots \mspace{14mu} \tau_{n}}} \\{{= {\rho \left( {s,t_{0},\varphi} \right)}},{\rho \left( {s,t_{1},\varphi} \right)},\ldots \mspace{14mu},{\rho \left( {s,t_{n},\varphi} \right)}} \\{{= {\rho \left( {s_{0},\varphi} \right)}},{\rho \left( {s_{1},\varphi} \right)},\ldots \mspace{14mu},{\rho \left( {s_{n},\varphi} \right)}}\end{matrix}$

Example 2

As an example of a computation graph, according to an aspect of thepresent disclosure, let s=(1, 0), (2, 0.1), (3, 0.2), (4, 0.3), (3,0.4), (3, 0.5) and q=(1,0), (2, 0.1), (3, 0.2), (4, 0.3), (5, 0.4), (6,0.5). Consider the specification ϕ=⋄□_([0,0.2])(s>4). For the timedtrace s, ϕ is not satisfied since the signal never exceeds four, while qdoes satisfy ϕ. For the robustness trace,

τ(s,t,ϕ)=−3,−2,−1,−1,−ρ_(max),−ρ_(max)

τ(q,t,ϕ)=−3,−2,−1,0,−ρ_(max),−ρ_(max)

FIG. 1(a) depicts a graphical representation of the STL formula ϕ fromExample 1. s represents the timed trace input signal, 4.0 represents theSTL parameter 104, and the blocks 106, 108 and 110 represent the STLoperators.

When t⊕I is beyond the length of the timed trace, the robustness isequal to −ρ_(max) because ϕ is not satisfied. Then the robustness may becomputed by taking the max over the robustness trace ρ(s, ϕ)=−1, ρ(q,ϕ)=0.

A natural extension to STL is parametric STL (pSTL). In pSTL, thespecifications are introduced as parametric templates. Typically, thetemplates are proposed by a user and the goal is to fit parameter valuesgiven a timed trace. The mapping from a timed trace to parameter valuesis a form of feature extraction on varying-length time series data.Feature extraction from time-series data, particularly ones of varyinglength, is challenging due to the different time scales, and difficultyin defining a similarity metric. Traditional methods such as dynamictime warping and the closely related Skorokhod metric are useful, butmay be inadequate in learning logical structure. Such methods may alsobe computationally intensive. After the feature extraction step, furtheranalysis, such as clustering or regression, may be applied on theextracted feature space.

Let ϕ_(P) be a specification template with parameters P (e.g.,ϕ_(α)=f(s)<α). Given a timed traces, a valuation v(P) may be sought suchthat ϕ_(v(P)) is the best description of s. This is equivalent tosolving an optimization problem defined by:

v′(P)=arg min ρ(s,t,ϕ)²  Eq. (1)

A solution according to one aspect of the disclosure may includeparameter values v(P) such that ρ(s, t, ϕ_(v(P))))=0. However, sincecalculating the robustness of an STL specification involves recursivelyapplying several max and min operations, solving Eq. (1) becomes anon-smooth, non-linear and non-convex problem. Standard optimizationtechniques such as gradient descent and genetic algorithms becomeineffective because gradients are non-smooth, and the robustness isexpensive to compute. While alternate solutions may exist, thosesolutions either rely on the formula being monotonic, which is notalways the case, or they use a Monte-Carlo sampling or recursivetechnique which can be very expensive to compute.

Aspects of the present disclosure present a solution to the problem ofdeveloping a smoothly differentiable computation graph that computes therobustness trace of a pSTL formula that contains □ (always) and ⋄(eventually) temporal operators. The set of logical formulas thatcontain only □ (always) and ⋄ (eventually) temporal operators arereferred to herein as the always-eventually fragment of STL. Though notcomplete, this fragment is expressive, and contains many specificationsof practical interest, including but not limited to the following:

-   -   (1) Safety: If ϕ is a logical formula that represents an unsafe        condition, then □        ϕ represents safety.    -   (2) Timely response: If ϕ is an event that the system must        respond to with an action ψ within some maximum interval I, the        requirement on the system can be expressed as ϕ⋄_(I)ψ. Although        the implication operator may not be explicitly included in the        grammar, it may be easily written in terms of the other Boolean        operators as        ϕ∨⋄_(I)ψ.    -   (3) Stability: Suppose ϕ represents a target region of        attraction. Then, stability of the system to this region can be        written as ⋄□ϕ.

Aspects of the present disclosure include conversion of a computationgraph into a smooth computation graph, such that it may bedifferentiated and its parameters learned through backpropagation. Thepresent disclosure provides an approach to transforming the parse treeof an STL formula into a computation graph and further how thecomputation graphs can be smoothly approximated. According to one aspectof the disclosure, an application of such methods may be made totest-case generation for autonomous driving scenarios.

Aspects of the present disclosure include systems and methods forconstructing computation graph representations of robustness androbustness trace calculations for STL formulas. FIG. 1b depicts ahigh-level flow diagram 150 representative of one such method. As shownin block 152, a timed trace, s, and an STL formula ϕ may be provided asinput to the system. As shown in block 154, the timed trace s may bereversed by reversing the order of elements in the trace. As shown inblock 156, a computation graph G may be constructed, as furtherexplained below, by combining G_(φ) _(i) , where φ_(i)∈O_(ϕ). Accordingto one aspect, the inputs of G_(φ) _(i) may only depend on outputs fromG_(ϕj), j>i. The overall graph may be combined by feeding the outputs ofthe graphs of the each subformula i into the graph of its parent formulaj. As shown in block 158, the computation graph G may be run by feedingthe reversed input sequence to the base of the computation graph andpropagating through to the parent graphs, until it reaches the top-mostoutput. The output of which, as shown in block 160, may be reversed toget a robustness trace, τ^(G)(s, t, ϕ), by reversing the number ofelements. From the robustness trace, a robustness degree, ρ^(G)(s, t,ϕ), may be determined by taking the first element of the robustnesstrace, as shown in block 162. This procedure will consistently produce acorrect output.

By propagating backwards in time via recurrent computation graphs, thesystem may compute the robustness and the robustness tracesimultaneously. This is particularly effective for formulas involvingnested temporal operators. A computation graph representation, G, ispresented for the predicate (f(s)<c), all the Boolean connectives(negation/not, conjunction/and, disjunction/or), and the eventually andalways temporal operators. The soundness of the method may also beproven in such an approach. For the Until and Then operators, the secondargument of the inner min operation may be □_([t;t′])ϕ and⋄_([t;t′]ϕ, respectively. Thus the robustness trace may be computed for each term in the robustness formula and the appropriate max and min functions may be taken to obtain the robustness trace for the Until and Then operations.)

The outputs of the temporal graph may be precisely the elements of therobustness trace but in reverse. This reversal is accounted for in themethod of FIG. 1 (block 110). While the method of FIG. 1, as shown anddescribed, outputs ρ (s, t, ϕ) (s; t;) and τ (s, t, ϕ), since we areusing a computation graph, the robustness and robustness trace of anysubformula of ϕ may be as the graph is propagated.

To construct the computation graph, according to one aspect of thedisclosure, the following definitions may be made:

-   -   Definition 5: (Valid Robustness Trace): C denotes a method for        computing the robustness of an STL formula ϕ, and τ^(C)(s, t, ϕ)        is t robustness trace of ϕ using method C. Then, τ^(C)(s, t, ϕ)        is a valid robustness trace for ϕ if τ^(C)(s, t, ϕ)=τ(s, t, ϕ).    -   Definition 6: (Soundness): C is sound if it produces a valid        robustness trace for all timed traces s and all STL formulas ϕ.

According to one aspect of the disclosure, the computation graph G maybe a directed acyclic graph made up of smaller computation graphs G_(φ)_(k) that take a robustness trace of the subformula as inputs, andoutputs the robustness trace after applying φ_(k).

Since G is a directed acyclic graph, according to one aspect, where eachsubgraph is a “node”, it may have a topological ordering. Thistopological ordering of G given ϕ may be precisely governed by O_(ϕ),the post-order traversal of the parse tree generated by ϕ.

Example 3

As another example of a computation graph, according to an aspect of thepresent disclosure, let θ=⋄□(ϕ∧ψ), then O_(θ)={ϕ, ψ, ∧, □, ⋄}. FIG. 2depicts a parse tree 202, PT_(θ), and computation graph 204, G_(θ). Thecomputation graph 204, G_(θ), of the formula θ may be constructed byconnecting the subgraphs whose nodes represent each operation. Thetopological ordering of G_(θ) may be the post-order traversal of theparse tree 202, PT_(θ), generated by ϕ. Therefore, the topologicalordering of G_(θ) may be G_(ϕ), {G_(ψ), G_(∧), G_(□), G_(⋄)}.

FIGS. 3(a)-(c) illustrate graphs, according to an aspect of the presentdisclosure, representing the robustness at a particular time t_(i),denoted by G_(φ) ^((i)), of the predicate, (f(s)<c) (FIG. 3(a)), andBoolean connectives: negation/not (FIG. 3(b)); conjunction/and,disjunction/or (FIG. 3(c)). Computing the robustness of these operatorsmay rely only on elementary operations, such as addition, subtraction,comparisons (e.g., “less than or equal”, “greater than”, or the like)and computing the maximum or minimum of a set of numbers. ConstructingG_(φ) ^((i)) may be performed on a case-by-case basis, depending on thelogical formula, as described herein.

To compute the robustness trace, i.e., construct the graph, G_(φ), G_(φ)^((i)) may be repeated over the timed trace. Using recurrent neuralnetworks and their ability to effectively process sequential data, arecurrent computation graph model may be used to compute the robustness,and robustness trace of the ⋄ (eventually) and □ (always) operators.

Suppose ψ=◯_(I)ϕ, where ◯ represents either ⋄ (eventually) or □(always). The graphical model for G_(◯) is depicted in FIG. 4 as arecurrent network for computing robustness of STL formulas with thetemporal operations ⋄ (eventually) or □ (always). G_(◯) takes arobustness trace of length N+1 as an input, and outputs a robustnesstrace after applying the ◯ operator. With imprecise notation, ρ_(i) mayrepresent ρ(s, t_(i), ϕ), h_(i) may represent the hidden state, ando_(i) may represent the output at each cell i. Since I describes futuretimes (i.e., 0≤a<b) from the given current time t, there is a need topropagate backwards in time in order for the computation graph to haveknowledge about the values at t⊕I. For ease of notation, the backwardssequence may be defined as {tilde over (ρ)}, i.e., ρ_(N)={tilde over(ρ)}₀, . . . ρ_(i)={tilde over (ρ)}_(N−1), . . . ρ₀={tilde over(ρ)}_(N).

If ◯=⋄, then h₀=−ρ_(max) and m=max; otherwise if ◯=□, h₀=ρ_(max) andm=min. The computation graph for the m operator is given in FIG. 3(c),but adjusted to accept more inputs. For each computation node (cell),the following equations may be applied:

Case 1: I=[0, ∞)

o=h _(i) =m({tilde over (ρ)}_(i) ,h _(i) ,h _(i+1) =o _(i)

Case 2: I=[a, ∞), a>0. The start of s may be truncated to obtain Case 1.

Case 3: I=[0, b], b<∞. Let m be the number of samples from s that lie in(0,b]. Then h_(i)∈R^(m),h_(i)=[h_(i) ₁ ,h_(i) ₂ , . . . h_(i) _(m) ], h₀is a vector of ±ρ_(max) instead of a scalar, and

o _(i) =m({tilde over (p)} _(i) ,h _(i)),h _(i+1)=[h _(i) ₂ , . . . h_(i) _(m) ,{tilde over (p)} _(i)]

Case 4: I=[a, b], b<∞. Let m be the number of samples from s that lie in(0, b], and k be the number of samples from s that lie in I. Thenh_(i)∈R^(m), h_(i)=[h_(i) ₁ , h_(i) ₂ , . . . h_(i) _(m) ] is a vectorof ±ρ_(max) instead of a scalar, and

o _(i) =m(h _(i) _(m−k+1) , . . . h _(i) _(m) )h _(i+1)=[h _(i) ₂ , . .. h _(i) _(m) ,{tilde over (ρ)}_(i)]

Equivalently, we can truncate the start of s to obtain Case 3.

For both Case 3 and 4, h_(i) may be a vector of all the robustnessvalues at each time step, t′ ∈t⊕I, excluding the value at the currenttime for that cell i. Then the h_(i+1) update may be a shift of thevector in time by one step, removing the oldest value, and updating thenewest values with the current value {tilde over (ρ)}_(i).

Example 4

As another example of a computation graph, according to an aspect of thepresent disclosure, Let ϕ=⋄_([0.2,0.3])s>2. Using s=(1,0), (2, 0.1), (3,0.2), (4, 0.3), (3, 0.4), (3, 0.5) from Example 2 above, the computationgraph G_(ϕ) is depicted in FIG. 5. The diamond node represents the ⋄(eventually) operator and uses the max operation. As shown, the outputsgive the reversed robustness trace, and ρ(s, t, ϕ)=2 is the last output.

Given the details and construction of G described above, the soundnessof the sequence τ^(G) (s, t, ϕ) may be proposed as the equivalent to therobustness trace τ (s, t, ϕ). Such a proposition may be proven bystructural induction on ϕ. As a base case, it may be shown that τ^(G)(s, t, f (s(t))<c) is the same as τ (s, t, f (s(t))<c). By construction(see FIG. 3(a)), ρ^(G) (s, t, f (s(t))<c)=c−f (s(t))=ρ(s, t, f(s(t))<c). Thus, τ^(G) (s, t, f (s(t))<c)=τ(s, t, f (s(t))<c).

For the induction step, it may be assumed that τ^(G)(s, t, ϕ) and τ^(G)(s, t, ψ) are valid robustness traces for ϕ and ψ respectively. Thismeans it may be assumed that ρ^(G) (s, t_(i), ϕ)=ρ(s, t_(i), ϕ) andρ^(G) (s,t_(i),ψ)=ρ(s, t_(i), ψ). It is needed to show that τ^(G)(s, t,ϕ∧ψ), τ^(G)(s, t, ϕ∨ψ), τ^(G)(s,t, ¬, ϕ), τ^(G)(s,t, ⋄ϕ) and τ^(G) (s,t, ⋄ϕ) are all valid robustness traces. Showing the cases for ∧, ∨ and ¬follows naturally from the construction of their computation graphs. Forclarity and brevity, only the proof for ∨ is given:

ρ^(G)(s, t_(i), φψ) = min (ρ^(G)(s, t_(i), φ), ρ^(G)(s, t_(i), ψ)) = min (ρ(s, t_(i), φ), ρ(s, t_(i), ψ)) = ρ(s, t_(i), φψ).

While only the proof for ∨ is given above, one skilled in the art willrecognize that the proofs for the cases for ∧ and ¬ follow similarly.

For ◯=⋄ or □, by construction, ρ^(G) (s, t_(i), ◯ϕ)=o_(n−i), since thetimed trace is fed backwards in time through the network, o_(i) dependson information in the future only, i.e., it is computed using thesubsequence at t_(N−i). Again, by construction on o_(i) and h_(i), forall the different cases for I described above:

$\begin{matrix}{o_{N - 1} = {\begin{matrix}m \\{k \in {t_{i} \oplus I}}\end{matrix}{\rho \left( {s,t_{k},\varphi} \right)}}} \\{= {\rho\left( {s,t_{i},} \right.}}\end{matrix}\therefore{\rho^{G}\left( {s,t_{i},{{\rho\left( {s,t_{i},} \right.}}} \right.}$

To prove for U (which follows to T, first a sequence from t₀ to t_(i)may be defined as (s, t⁻¹). As it has been proven that other operatorsmay all have valid robustness traces, and by construction:

$\begin{matrix}{{\rho^{G}\left( {s,t,{\varphi \; U_{I}\psi}} \right)} = {\max\limits_{i \in {t \oplus I}}\left( {\min \left( {{\rho^{G}\left( {s,t,\psi} \right)},{\min\limits_{i^{\prime} \in {\lbrack{0,i}\rbrack}}{\rho^{G}\left( {s,t_{i^{\prime \;}},\varphi} \right)}}} \right)} \right)}} \\{= {\max\limits_{i \in {t \oplus I}}\left( {\min \left( {{\rho^{G}\left( {s,t,\psi} \right)},{\rho^{G}\left( {s,t_{- i^{\;}},{\square\varphi}} \right)}} \right)} \right)}} \\{= {\max\limits_{i \in {t \oplus I}}\left( {\min \left( {\rho^{G}\left( {s,t,\left( {{\rho^{G}\left( {s,t_{i^{\;}},\psi} \right)}{\rho^{G}\left( {s,{t_{- i}\varphi}} \right)}} \right)} \right)} \right)} \right.}} \\{= {\max\limits_{i \in {t \oplus I}}\left( {\min \left( {{\rho^{G}\left( {s,t_{i},\psi} \right)},{\min\limits_{i^{\prime} \in {\lbrack{0,i}\rbrack}}{\rho \left( {s,t_{i^{\prime \;}},\varphi} \right)}}} \right)} \right)}} \\{= {\rho \left( {s,t,{\varphi \; U_{I}\psi}} \right)}}\end{matrix}$

By using τ^(G) (s, t_(−i), ϕ) instead of ρ^(G)(s, t_(−i), ϕ) therobustness trace (up to time t_(i)) for ρ^(G) (s, t, ϕU_(I)ψ) may befound.

Thus, it may be shown that G can compute a valid robustness trace forany STL formula. In particular, the computation complexity of G can beseen to scale linearly with the length of the signal, or quadraticallyfor the Until and Then operations, and also linearly as more temporaloperations (i.e., tree depth of PT) are applied.

As detailed herein, a computation graph may be used to compute therobustness of a STL formula exactly. Taking gradients, however, withrespect to pSTL parameters may be difficult due to the non-smooth natureof the max and min functions. By making a smooth approximation, ageneral computation graph library may be used to take the gradient withrespect to a parameter using backpropagation.

FIG. 6 depicts an illustrative method for solving for pSTL parameters ofEq. (1), above, using a computation graph, G and differentiableapproximations. The method of FIG. 6 may be a gradient descentalgorithm, where the termination criteria depends on the true value of pand the gradient, and the relaxation parameter, w, is annealed over eachiteration such that towards the end of the optimization process, theapproximation of the max and min functions are more exact.

The illustrative method may take a timed trace s, pSTL formula ϕ_(P), amaximum number of iterations N, a robustness tolerance ρ_(tol), agradient tolerance g_(tol), an annealing function A, and a learning rateγ as inputs. The method 600 may output the valuation v(P). As shown inblock 605 the method may begin by initializing the valuation v(P),followed by constructing the computation graph G, as shown in block 610and described above. As shown in block 615, a “for” loop may beimplemented such that for every i←0 up to a maximum N, ρ_(e)=ρ(s, t,ϕ_(v(P)))² is computed, as shown in block 620. If i←0 is greater than N,the method may terminate, as shown in block 655. As shown in block 625,the method 600 may determine if ρ_(e)<ρ_(tol). If ρ_(e)<ρ_(tot) (i.e.,ρ_(e) is less than a robustness tolerance), the method 600 mayterminate. If ρ_(e) is greater than the robustness tolerance, ρ_(tol),as shown in block 630, a scaling parameter may be determined from theannealing function w_(i)←A(i). As shown in block 635, a loss value maybe determined as

=ρ(s, t,ϕ_(v(P)); w_(i))². Backpropogation may be used on l, as shown inblock 640 to determine the differentiation,

$\frac{\partial }{\partial P}.$

As shown in block 645, if

${{\frac{\partial }{\partial P}}^{2} < _{tol}},$

the method 600 may terminate. If

${{\frac{\partial }{\partial P}}^{2} \geq _{tol}},$

v(P) may be updated using a step size of γ (the learning rate), as shownin block 650 and the method 600 may return to block 615.

According to one aspect of the disclosure, machine-learning software maybe implemented. Advantageously, the gradients may therefore be computedby using an auto-differentiation functionality. The method shown in FIG.6 demonstrates how we can iteratively tighten the approximation toobtain better and better precision. The illustrative method of may solveEq. (1) more efficiently than other known methods, such as directmethods and gradient descent via finite differencing.

According to one aspect of the disclosure, depending on the applicationand assumptions made herein, absolute exactness of the parameters is notstrictly necessary. For example, when pSTL is used for featureextraction, rather than for formal verification and/or model checking,post-analysis methods such as clustering or regression have inherentnoise assumptions which can be accounted for by the inexactness. Thus,using differentiable approximations (i.e., computing the gradient of acomputation with respect to its input variables) of the min and maxoperators in order may be used to make solving Eq. (1), above, moretractable. This differentiable approximation, when coupled with thecomputation graph representation G may offer the following benefits:

(1) backpropagation may be used to compute gradients, thus Eq. (1) canbe solved efficiently;

(2) a smooth differentiable approximation may improve convergence formore complicated pSTL formulas;

(3) the benefits of modern machine learning tools, such asstate-of-the-art auto-differentiation tools, may be leveraged. Forexample, optimization toolkits may be used, parallel hardware may beimplemented, and multiple signals may be evaluated via hatching.

Such an approximation technique may be applicable to spatial parameters,and may not apply to temporal parameters (i.e., it may not apply toparameters that define the interval I).

Let x∈

^(n) and w∈

_(≥0), then the max and min approximations are:

$\begin{matrix}{{{\left( {x;w} \right)} = \frac{\sum\limits_{i}^{n}{x_{i}{\exp \left( {wx}_{i} \right)}}}{\sum\limits_{j}^{n}{\exp \left( {wx}_{i} \right)}}},} & {{Eq}.\mspace{14mu} (2)} \\{{\left( {x;w} \right)} = \frac{\sum\limits_{i}^{n}{x_{i}{\exp \left( {- {wx}_{i}} \right)}}}{\sum\limits_{j}^{n}{\exp \left( {- {wx}_{i}} \right)}}} & {{Eq}.\mspace{14mu} (3)}\end{matrix}$

w is a scaling parameter, and when w→∞, this approximation may approachthe true maximum or minimum, while w=0 gives the average of the valuesin x. Alternate approaches may exist because this approximation becomesill-defined when w→∞, however, w may be restricted from getting toolarge (this parameter can be annealed) and thus ensure that x is scaledappropriately.

For every max and min operation in the robustness formula, it may bereplaced by

(Eq. (2)) and

(Eq. (3)) As Eq (2) and Eq. (3) use elementary operations, it may beconverted into a computation graph. As such, ρ(s, t, ϕ, w) may bewritten to indicate that the robustness is computed using theapproximation parameterized by w.

According to one aspect of the present disclosure, the methods describedherein may be applied to systematic scenario generation for autonomousdriving, such as a traffic-weaving scenario. In order to certify thesafety of a cyber-physical system, such as an autonomous vehicle, thesystem must be strenuously stress-tested in a diverse range of scenariosto ensure the system is certifiably safe and robust (i.e., that systemfailures occur within some ϵ-probability). For instance, autonomousvehicles must be tested for millions of miles on diverse scenarios. Evenwith simulation, such testing may be impractical and may hinder thedevelopment of the technology. For this reason, when developing a testsuite for cyber-physical systems, it is important to have a strongunderstanding of the factors that make a testing scenario easy ordifficult to pass.

While it is possible to generate test cases from simulation orreal-world test data where a failure occurred, it may be more valuableto use the data to inform the search for unseen scenarios that maystress the system in different ways. Given a large dataset, however, itmay be difficult to find structure and explainability in the data. Givena dataset of vehicles operating on the road, it may be difficult ingeneral to cluster time-series data, but even more difficult to explainthe behavior of a vehicle, such as “the car would turn left only ifthere was a sufficiently large gap.” A designer may have a vagueintuition of which quantities are important to monitor throughout thetask, but may not have a strong grasp of how exactly those quantitiesinteract with each other to produce interesting test cases.

According to one aspect of the disclosure, the problem of generatingtest cases for an autonomous vehicle in a traffic weaving setting may beconsidered. Such a scenario may be a challenging task in which twovehicles must change or swap lanes in a short amount of time, emulatinga highway on/offramp. One aspect of the present disclosure aims to usepSTL formulas to identify factors that contribute to makingtraffic-weaving challenging, and useful in designing challenging drivingscenarios.

FIG. 7 depicts a graphical representation of a traffic weaving scenario.In such a scenario, two cars must swap lanes in a short amount of time,emulating a highway on/off-ramp, for example. As shown, the merging car,or ego car, 704 may be attempting to merge onto the roadway 702, thecontinuing cars 706 may stay on the roadway and be used to populate theroad network. The aim, according to one aspect, is to generate drivingscenarios with varying difficult levels in order to develop a suite ofregression tests that track the progress of the autonomous drivingstack.

According to an aspect of the present disclosure, pSTL may be used tolearn “explanatory factors” in order to gain insight into why a drivingscenario may be challenging. The knowledge of these factors may beleveraged create a variety of driving scenarios with varying difficultyto stress-test an autonomous driving stack.

According to one aspect, a “time to collision” (ttc), (the estimatedtime between an event occurring and a potential collision of thevehicle) may be determined to be an important quantity to monitor whilethe vehicle executes a lane change. Ttc may be defined as:

$\begin{matrix}{{{ttc}\left( {x_{1},x_{2},v_{1},v_{2}} \right)} = {- \frac{x_{1} - x_{2}}{v_{1} - v_{2}}}} & {{Eq}.\mspace{14mu} (4)}\end{matrix}$

Here, x_(i) and v_(i) represent the longitudinal distance and velocityof a vehicle i, assuming both vehicles are traveling almost parallel.While human intuition may propose ttc as an interesting quantity tomonitor, aspects of the present disclosure seek to systematicallyquantify how ttc actually affects the difficulty of the scenario. Assuch, the following pSTL, formula is proposed:

ϕ=⋄□_([0,τ])(Δy<3∧ttc>γ)  Eq. (5)

where Δy is the lateral distance between an ego vehicle and anotherother vehicle. This formula, Eq. (5), may be translated as “eventually,when the ego car 704, is starting to move over to the adjacent lane overan interval of τ seconds, the ttc with the adjacent continuing car 706should be larger than γ.” The distance between the ego vehicle 704 andthe continuing vehicle 706 may be chosen to be greater than three (Δy<3)because the width of an average car is slightly less than three meters.The parameters to be learned are τ and γ. Since τ represents the lengthof the lane-change maneuver (when Δy<3), τ may be computed from obtaineddata.. Thus, it remains to find γ using the computation graph methodoutlined above. That is, given a dataset D, for each trial s^((i))∈D,the pSTL equation, Eq. (1), may be solved using the computation graphmethod presented herein to obtain {τ^((i))}_(i=1) ^(n) and{γ^((i))}_(i=1) ^(n).

According to one aspect, roughly 3000 traffic-weaving trials may begenerated using a road traffic simulation ckage, for example, theSimulation of Urban Mobility (SUMO). The simulation package may includea car-following and lane-changing model, as represented in the snapshotof FIG. 7. The formula may be applied to the front adjacent and rearadjacent cars of the ego vehicle. The adjacent side is defined as theside that the ego vehicle is attempting to change into. The results forτ^((i)) and γ^((i)) in both front and rear cases are shown in FIGS.8(a)-(b). Two cars are in collision if their distance from each other iswithin some tolerance. It may be seen that there is a distinct range 802(FIG. 8(a)) for γ that relates to front-end collisions γ∈[−2, 2]) andsimilarly, there is also a range 804 (FIG. 8(b)) that relates torear-end collisions γ∈[−4, 2]). These ranges correspond to small τ. Thismay indicate that when cars are changing lanes rapidly and have ttcwithin some threshold, those vehicles are more prone to collisions.There does not seem to be any relationship, however, when τ is larger,corresponding to a longer lane change.

These parameter bounds may be leveraged to design other vehicles on theroad to be more adversarial and hence “stresstest” the ego-car's drivingpolicy and accelerate the evaluation of an autonomous vehicle. Forexample, one can design a car that purposely drives to keep ttc belowsome threshold.

Aspects of the methods described herein show how parameters of logicalformulas can be trained with machine learning tools. The systems andmethods described herein embed logical structure into machine learningmodels.

FIG. 9 is a diagram illustrating an example of a hardware implementationfor a driving scenario generation system 900 according to aspects of thepresent disclosure. The scenario generation system 900 may be acomponent of a simulator or other testing environment, a vehicle, arobotic device, or other device. For example, as shown in FIG. 9, thescenario generation system 900 is a component of a car 928. Of course,aspects of the present disclosure are not limited to the scenariogeneration system 900 being a component of the car 928, as otherdevices, such as a simulator, bus, boat, drone, or robot, are alsocontemplated for using the scenario generation system 900.

The scenario generation system 900 may be implemented with a busarchitecture, represented generally by a bus 930. The bus 930 mayinclude any number of interconnecting buses and bridges depending on thespecific application of the scenario generation system 900 and theoverall design constraints. The bus 930 links together various circuitsincluding one or more processors and/or hardware modules, represented bya processor 920, a communication module 922, a location module 918, asensor module 902, a locomotion module 926, a planning module 924, and acomputer-readable medium 914. The bus 930 may also link various othercircuits such as timing sources, peripherals, voltage regulators, andpower management circuits, which are well known in the art, andtherefore, will not be described any further.

The scenario generation system 900 includes a transceiver 916 coupled tothe processor 920, the sensor module 902, a scenario generator 908, thecommunication module 922, the location module 918, the locomotion module926, the planning module 924, and the computer-readable medium 914. Thetransceiver 916 is coupled to antennae 934. The transceiver 916communicates with various other devices over a transmission medium. Forexample, the transceiver 916 may receive commands via transmissions froma user or a remote device. As another example, the transceiver 916 maytransmit driving statistics and information from the scenario generator908 to a server (not shown).

The scenario generation system 900 includes the processor 920 coupled tothe computer-readable medium 914. The processor 920 performs processing,including the execution of software stored on the computer-readablemedium 914 providing functionality according to the disclosure. Thesoftware, when executed by the processor 920, causes the scenariogeneration system 900 to perform the various functions described for aparticular device, such as the car 928, or any of the modules 902, 908,914, 916, 918, 920, 922, 924, 926. The computer-readable medium 914 mayalso be used for storing data that is manipulated by the processor 920when executing the software.

The sensor module 902 may be used to obtain measurements via differentsensors, such as a first sensor 906 and a second sensor 904. The firstsensor 906 may be a vision sensor, such as a stereoscopic camera or anRGB camera, for capturing 2D images. The second sensor 904 may be aranging sensor, such as a light detection and ranging (LIDAR) sensor ora radio detection and ranging (RADAR) sensor. Of course, aspects of thepresent disclosure are not limited to the aforementioned sensors asother types of sensors, such as, for example, thermal, sonar, and/orlasers are also contemplated for either of the sensors 904, 906. Themeasurements of the first sensor 906 and the second sensor 904 may beprocessed by one or more of the processor 920, the sensor module 902,the artificial neural network module 908, the communication module 922,the location module 918, the locomotion module 926, the planning module924, in conjunction with the computer-readable medium 914 to implementthe functionality described herein.

As previously discussed, the measurements from the first sensor 906 maybe used to capture 2D images. Furthermore, the measurements from thesecond sensor 904 may be used for depth measurements. In oneconfiguration, the data captured by the first sensor 906 and the secondsensor 904 may be transmitted to an external device via the transceiver916. The first sensor 906 and the second sensor 904 may be coupled tothe car 928 or may be in communication with the car 928.

The location module 918 may be used to determine a location of the car928. For example, the location module 918 may use a global positioningsystem (GPS) to determine the location of the car 928. The communicationmodule 922 may be used to facilitate communications via the transceiver916. For example, the communication module 922 may be configured toprovide communication capabilities via different wireless protocols,such as WiFi, long term evolution (LTE), 3G, etc. The communicationmodule 922 may also be used to communicate with other components of thecar 928 that are not modules of the scenario generation system 900.

The locomotion module 926 may be used to facilitate locomotion of thecar 928. As an example, the locomotion module 926 may control movementof the wheels. As another example, the locomotion module 926 may be incommunication with a power source of the car 928, such as an engine orbatteries. Of course, aspects of the present disclosure are not limitedto providing locomotion via wheels and are contemplated for other typesof components for providing locomotion, such as propellers, treads,fins, and/or jet engines.

The scenario generation system 900 also includes a planning module 924for planning a trajectory or controlling the locomotion of the car 928,via the locomotion module 926, based on the analysis performed by theartificial neural network 908. In one configuration, the planning module924 overrides the user input when the user input is expected (e.g.,predicted) to cause a collision. The modules may be software modulesrunning in the processor 920, resident/stored in the computer-readablemedium 914, one or more hardware modules coupled to the processor 920,or some combination thereof.

The scenario generator 908 may be in communication with the sensormodule 902, the transceiver 916, the processor 920, the communicationmodule 922, the location module 918, the locomotion module 926, theplanning module 924, and the computer-readable medium 914. In oneconfiguration, the scenario generator 908 receives sensor data from thesensor module 902. The sensor module 902 may receive the sensor datafrom the first sensor 906 and the second sensor 904. According toaspects of the disclosure, the sensor module 902 may filter the data toremove noise, encode the data, decode the data, merge the data, extractframes, or perform other functions. In an alternate configuration, thescenario generator 908 may receive sensor data directly from the firstsensor 906 and the second sensor 904.

The scenario generator 908 may define or receive one or more pSTLformula describing an operation, event, or action of the vehicle. ThepSTL formulas may include one or more parameters to be learned. Thescenario generator 908 may calculate or otherwise determine the pSTLparameters by using the backpropogation and computation graphs, asdescribed herein. The parameters, pSTL formulas may be used to generatestress testing scenarios and incorporated into autonomous vehicledriving behaviors.

As shown in FIG. 9, the scenario generator 908 may include an artificialneural network 909 (e.g., machine learning module) may include a featureextractor 910, a classifier 912, and a predictor 932. The featureextractor 910, the classifier 912, and the predictor 932 may becomponents of a same or different convolutional neural network (CNN),such as a deep CNN. The artificial neural network 908 is not limited toa CNN and may be another type of artificial neural network, such as asupport vector machine (SVM). The feature extractor 910 receives a datastream from a data source, such as the first sensor 906 and the secondsensor 904, or a data storage device (not shown). The data stream may bedata combined from the first sensor 906 and the second sensor 904. Forexample, the data stream may be a 2D RGB image from the first sensor 906that is merged with LIDAR data points from the second sensor 904. Inanother configuration, the data stream is a separate stream from eachsensor 904, 906. The data stream may include streams such as timedtraces. The feature extractor 910 extracts (e.g., identifies) parametersof the data stream. The artificial neural network 908 may include one ormore feature extractors 910. For example, one feature extractor 910 maybe a parameter identifier of a pSTL formula as detailed above.

The predictor 932 may receive the output from the classifier 912 topredict a behavior of an observed agent, such as a time to collision ofa merging vehicle, or other behavioral action. The prediction may bebased on training. Additionally, the prediction may be based onclassified time traces and determined robustness degrees and values.Based on the predicted behavior, the predictor 932 may also estimate aset of behavioral constraints of the agent. In another configuration,the artificial neural network 908 may output the predicted behaviordetermined by the predictor 932 to one or more of the sensor module 902,the transceiver 916, the processor 920, the communication module 922,the location module 918, the locomotion module 926, the planning module924, and the computer-readable medium 914.

For example, the annotated data may output to the processor 920 toretrieve a set of behavioral constraints that correspond to thepredicted behavior from the computer-readable medium 914 or planningmodule 924. Additionally, based on the set of behavioral constraints forthe agent, one or more of the modules 902, 908, 914, 916, 918, 920, 922,924, 926 may determine or update a set of behavioral constraints of thecar 928. The planning module 924 may perform motion planning,decision-making, or other vehicular control functionality based on theset of behavioral constraints of the car 928.

Based on the teachings, one skilled in the art should appreciate thatthe scope of the present disclosure is intended to cover any aspect ofthe present disclosure, whether implemented independently of or combinedwith any other aspect of the present disclosure. For example, anapparatus may be implemented or a method may be practiced using anynumber of the aspects set forth. In addition, the scope of the presentdisclosure is intended to cover such an apparatus or method practicedusing other structure, functionality, or structure and functionality inaddition to, or other than the various aspects of the present disclosureset forth. It should be understood that any aspect of the presentdisclosure may be embodied by one or more elements of a claim.

The word “exemplary” is used herein to mean “serving as an example,instance, or illustration.” Any aspect described herein as “exemplary”is not necessarily to be construed as preferred or advantageous overother aspects.

Although particular aspects are described herein, many variations andpermutations of these aspects fall within the scope of the presentdisclosure. Although some benefits and advantages of the preferredaspects are mentioned, the scope of the present disclosure is notintended to be limited to particular benefits, uses or objectives.Rather, aspects of the present disclosure are intended to be broadlyapplicable to different technologies, system configurations, networksand protocols, some of which are illustrated by way of example in thefigures and in the following description of the preferred aspects. Thedetailed description and drawings are merely illustrative of the presentdisclosure rather than limiting, the scope of the present disclosurebeing defined by the appended claims and equivalents thereof.

As used herein, the term “determining” encompasses a wide variety ofactions. For example, “determining” may include calculating, computing,processing, deriving, investigating, looking up (e.g., looking up in atable, a database or another data structure), ascertaining and the like.Additionally, “determining” may include receiving (e.g., receivinginformation), accessing (e.g., accessing data in a memory) and the like.Furthermore, “determining” may include resolving, selecting, choosing,establishing, and the like.

As used herein, a phrase referring to “at least one of” a list of itemsrefers to any combination of those items, including single members. Asan example, “at least one of: a, b, or c” is intended to cover: a, b, c,a-b, a-c, b-c, and a-b-c.

The various illustrative logical blocks, modules and circuits describedin connection with the present disclosure may be implemented orperformed with a processor specially configured to perform the functionsdiscussed in the present disclosure. The processor may be a neuralnetwork processor, a digital signal processor (DSP), an applicationspecific integrated circuit (ASIC), a field programmable gate arraysignal (FPGA) or other programmable logic device (PLD), discrete gate ortransistor logic, discrete hardware components or any combinationthereof designed to perform the functions described herein.Alternatively, the processing system may comprise one or moreneuromorphic processors for implementing the neuron models and models ofneural systems described herein. The processor may be a microprocessor,controller, microcontroller, or state machine specially configured asdescribed herein. A processor may also be implemented as a combinationof computing devices, e.g., a combination of a DSP and a microprocessor,a plurality of microprocessors, one or more microprocessors inconjunction with a DSP core, or such other special configuration, asdescribed herein.

The steps of a method or algorithm described in connection with thepresent disclosure may be embodied directly in hardware, in a softwaremodule executed by a processor, or in a combination of the two. Asoftware module may reside in storage or machine readable medium,including random access memory (RAM), read only memory (ROM), flashmemory, erasable programmable read-only memory (EPROM), electricallyerasable programmable read-only memory (EEPROM), registers, a hard disk,a removable disk, a CD-ROM or other optical disk storage, magnetic diskstorage or other magnetic storage devices, or any other medium that canbe used to carry or store desired program code in the form ofinstructions or data structures and that can be accessed by a computer.A software module may comprise a single instruction, or manyinstructions, and may be distributed over several different codesegments, among different programs, and across multiple storage media. Astorage medium may be coupled to a processor such that the processor canread information from, and write information to, the storage medium. Inthe alternative, the storage medium may be integral to the processor.

The methods disclosed herein comprise one or more steps or actions forachieving the described method. The method steps and/or actions may beinterchanged with one another without departing from the scope of theclaims. In other words, unless a specific order of steps or actions isspecified, the order and/or use of specific steps and/or actions may bemodified without departing from the scope of the claims.

The functions described may be implemented in hardware, software,firmware, or any combination thereof. If implemented in hardware, anexample hardware configuration may comprise a processing system in adevice. The processing system may be implemented with a busarchitecture. The bus may include any number of interconnecting busesand bridges depending on the specific application of the processingsystem and the overall design constraints. The bus may link togethervarious circuits including a processor, machine-readable media, and abus interface. The bus interface may be used to connect a networkadapter, among other things, to the processing system via the bus. Thenetwork adapter may be used to implement signal processing functions.For certain aspects, a user interface (e.g., keypad, display, mouse,joystick, etc.) may also be connected to the bus. The bus may also linkvarious other circuits such as timing sources, peripherals, voltageregulators, power management circuits, and the like, which are wellknown in the art, and therefore, will not be described any further.

The processor may be responsible for managing the bus and processing,including the execution of software stored on the machine-readablemedia. Software shall be construed to mean instructions, data, or anycombination thereof, whether referred to as software, firmware,middleware, microcode, hardware description language, or otherwise.

In a hardware implementation, the machine-readable media may be part ofthe processing system separate from the processor. However, as thoseskilled in the art will readily appreciate, the machine-readable media,or any portion thereof, may be external to the processing system. By wayof example, the machine-readable media may include a transmission line,a carrier wave modulated by data, and/or a computer product separatefrom the device, all which may be accessed by the processor through thebus interface. Alternatively, or in addition, the machine-readablemedia, or any portion thereof, may be integrated into the processor,such as the case may be with cache and/or specialized register files.Although the various components discussed may be described as having aspecific location, such as a local component, they may also beconfigured in various ways, such as certain components being configuredas part of a distributed computing system.

The machine-readable media may comprise a number of software modules.The software modules may include a transmission module and a receivingmodule. Each software module may reside in a single storage device or bedistributed across multiple storage devices. By way of example, asoftware module may be loaded into RAM from a hard drive when atriggering event occurs. During execution of the software module, theprocessor may load some of the instructions into cache to increaseaccess speed. One or more cache lines may then be loaded into a specialpurpose register file for execution by the processor. When referring tothe functionality of a software module below, it will be understood thatsuch functionality is implemented by the processor when executinginstructions from that software module. Furthermore, it should beappreciated that aspects of the present disclosure result inimprovements to the functioning of the processor, computer, machine, orother system implementing such aspects.

If implemented in software, the functions may be stored or transmittedover as one or more instructions or code on a computer-readable medium.Computer-readable media include both computer storage media andcommunication media including any storage medium that facilitatestransfer of a computer program from one place to another.

Further, it should be appreciated that modules and/or other appropriatemeans for performing the methods and techniques described herein can bedownloaded and/or otherwise obtained by a user terminal and/or basestation as applicable. For example, such a device can be coupled to aserver to facilitate the transfer of means for performing the methodsdescribed herein. Alternatively, various methods described herein can beprovided via storage means, such that a user terminal and/or basestation can obtain the various methods upon coupling or providing thestorage means to the device. Moreover, any other suitable technique forproviding the methods and techniques described herein to a device can beutilized.

It is to be understood that the claims are not limited to the preciseconfiguration and components illustrated above. Various modifications,changes, and variations may be made in the arrangement, operation, anddetails of the methods and apparatus described above without departingfrom the scope of the claims.

What is claimed is:
 1. A method of evaluating a test case comprising:receiving a timed trace input; generating a parametric signal temporallogic formula representative of a driving scenario; solving theparametric temporal logic formula for at least one parameter, whereinthe value of the at least one parameter provides an optimal descriptionof the timed trace; and evaluating the test case using the value of theat least one parameter.
 2. The method of claim 1 wherein solving theparametric temporal logic formula comprises: reversing the timed trace;constructing a computation graph from the parametric signal temporallogic formula and the reversed timed trace; running the computationgraph to generate an output; and reversing the output to determine arobustness trace of the signal temporal logic formula.
 3. The method ofclaim 2 further comprising generating a differentiable approximation ofat least one of a minimum and maximum operator.
 4. The method of claim 3wherein the generation of the differentiable approximation comprisesbackpropogation to compute a gradient of the parametric signal temporallogic formula.
 5. The method of claim 1 further comprising determining arobustness degree from the robustness trace, the robustness degreerepresentative of a quantification of the timed trace satisfying theparametric signal temporal logic formula.
 6. The method of claim 5,wherein the robustness degree is the first element of the robustnesstrace.
 7. The method of claim 1 further comprising optimizing therobustness trace using machine learning for feature extraction.
 8. Themethod of claim 1 wherein the robustness trace comprises a robustnessvalue of the timed trace.
 9. The method of claim 1 further comprisinggenerating a test case for an autonomous vehicle, the parametric signaltemporal logic formula representative of a driving scenario.
 10. Themethod of claim 9 wherein the driving scenario is a lane-changingoperation.
 11. The method of claim 10 wherein the lane-changingoperation comprises a time-to-collision quantity.
 12. The method ofclaim 11 where in the parametric signal temporal logic formula includesa time-to-change-lane internal parameter and a time-to-collisionparameter.
 13. A system for evaluating a testing scenario, the systemcomprising: a memory; and at least one processor coupled to the memory,the at least one processor configured: receive a timed trace input;generate a parametric signal temporal logic formula representative of adriving scenario; solve the parametric temporal logic formula for atleast one parameter, wherein the value of the at least one parameterprovides an optimal description of the timed trace; and evaluate thetest case using the value of the at least one parameter.
 14. The systemof claim 13 wherein the processor is further configured to: reverse thetimed trace; construct a computation graph from the parametric signaltemporal logic formula and the reversed timed trace; run the computationgraph to generate an output; and reverse the output to determine arobustness trace of the signal temporal logic formula.
 15. The system ofclaim 14 wherein the processor is configured to generate adifferentiable approximation of at least one of a minimum and maximumoperator.
 16. The system of claim 15 wherein the processor is configuredto generate the differentiable approximation using backpropogation tocompute a gradient of the parametric signal temporal logic formula. 17.The system of claim 14 wherein the processor is configured to generate arobustness degree from the robustness trace, the robustness degreerepresentative of a quantification of the timed trace satisfying theparametric signal temporal logic formula.
 18. The system of claim 17,wherein the robustness degree is the first ent of the robustness trace.19. A non-transitory computer-readable medium having program coderecorded thereon for evaluating a testing scenario, the program codeexecuted by a processor and comprising: program code to receive a timedtrace input; program code to generate a parametric signal temporal logicformula representative of a driving scenario; program code to solve theparametric temporal logic formula for at least one parameter, whereinthe value of the at least one parameter provides an optimal descriptionof the timed trace; and program code to evaluate the test case using thevalue of the at least one parameter.
 20. The non-transitorycomputer-readable medium of claim 19 further comprising: program code toreverse the timed trace; program code to construct a computation graphfrom the parametric signal temporal logic formula and the reversed timedtrace; program code to run the computation graph to generate an output;and program code to reverse the output to determine a robustness traceof the signal temporal logic formula.